Image sensor

ABSTRACT

An imaging device may code light, passing through an imaging optical lens arranged in a multi-lens array (MLA), and may transmit the light to a sensing element, and the sensing element may restore an image based on sensed information.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No.10-2020-0055588, filed on May 11, 2020, and Korean Patent ApplicationNo. 10-2020-0143664, filed on Oct. 30, 2020, in the Korean IntellectualProperty Office, the entire disclosures of which are herein incorporatedby reference for all purposes.

BACKGROUND 1. Field

Methods and apparatuses consistent with example embodiments of thedisclosure are related to sensing an image.

2. Description of Related Art

Due to the development of optical technologies and image processingtechnologies, image capturing devices are being utilized in a wide rangeof fields, for example, multimedia content, security or recognition. Forexample, an image capturing device may be mounted in a mobile device, acamera, a vehicle or a computer, to capture an image, to recognize anobject, to acquire data for controlling a device, or the like. A volumeof the image capturing device may be determined based on, for example, asize of a lens, a focal length of a lens, and a size of a sensor. If thesize of the lens decreases, the focal length of the lens may decrease.To reduce the volume of the image capturing device, a multi-lensincluding compact and small lenses may be used.

SUMMARY

One or more example embodiments may address at least the above problemsand/or disadvantages and other disadvantages not described above. Also,the example embodiments are not required to overcome the disadvantagesdescribed above, and an example embodiment may not overcome any of theproblems described above.

According to an aspect of an example embodiment, provided is an imagesensor including: a mask array including a plurality of mask elements,the plurality of mask elements being configured to, among light passingthrough imaging optical lenses and incident onto the mask array in aplurality of directions, block light in a first portion of the pluralityof directions, and allow light in a second portion of the plurality ofdirections to pass therethrough; and a sensing array including aplurality of sensing elements, the plurality of sensing elements beingconfigured to sense the light passing through the imaging optical lensesand the mask array.

The image sensor may further include: a color filter provided above thesensing array and configured to filter light of a portion of wavelengthbands from light incident on each of the plurality of sensing elements,wherein the mask array is provided between the color filter and thesensing array.

The image sensor may further include: a condensing lens array providedabove the sensing array, wherein the mask array is provided between thecondensing lens array and the sensing array.

The mask array and the plurality of sensing elements may be spaced apartfrom each other by 1 micrometer (μm) or less.

The mask array and the plurality of sensing elements may be in contactwith each other.

A first region of the mask array corresponding to a sensing element ofthe plurality of sensing elements may include: an aperture regionoccupying an area corresponding to an aperture ratio with respect to atotal area of the first region; and a masked region occupying aremaining area of the first region, the plurality of mask elements beingprovided in the masked region.

The aperture ratio may be between about 40% and about 60%.

In each partial region of the mask array, an area occupied by anaperture nay be greater than or equal to an area corresponding to a setaperture ratio.

The mask array may be segmented into a plurality of group regionscorresponding to a plurality of sensing element groups, and each of theplurality of group regions in the mask array may be configured to covera sensing element group, the sensing element group including a pluralityof sensing elements that are grouped to represent a single pixel.

A masking pattern of a group region may be repeated in the mask array.

All of the plurality of group regions in the mask array may have a samemasking pattern.

A number of spaces included in a masking pattern, which is repeated inthe mask array, may be greater than or equal to a number of the imagingoptical lenses.

The plurality of mask elements may have two or more transmission levels.

Each of the plurality of mask elements may be segmented into a pluralityof regions, and a transmittance of each of the plurality of maskelements may be determined based on a ratio of an open region and aclosed region among the plurality of regions.

The image sensor may further include: a processor configured to restorean image based on sensing information sensed by the plurality of sensingelements.

The processor may be further configured to generate frequencyinformation by transforming the sensing information to a frequencydomain, to generate deblurred frequency information by dividing thefrequency information by a frequency conversion result of a blur kernel,the blur kernel corresponding to a masking pattern of the mask array,and to restore a high-resolution image by inversely transforming thedeblurred frequency information to a time domain.

The mask array may include a plurality of masking patterns, and each ofthe plurality of masking patterns may be configured to cover a sensingelement group, the sensing element group including two or more sensingelements in the sensing array.

According to an aspect of an example embodiment, provided a cameradevice including: an imaging lens array including imaging opticallenses, the imaging optical lenses configured to transmit light receivedfrom an outside of the camera device; a sensing array including aplurality of sensing elements, the plurality of sensing elements beingconfigured to sense light passing through the imaging lens array; and amask array including a plurality of mask elements, the mask array beingprovided between the imaging lens array and the sensing array.

The mask array may be provided at one of a position inside the sensingarray and a position in contact with the sensing array on the pluralityof sensing elements.

A first region of the mask array corresponding to a sensing element ofthe plurality of sensing elements may include: an aperture regionoccupying an area corresponding to an aperture ratio with respect to atotal area of the first region; and a masked region occupying aremaining area in the first region, the plurality of mask elementsprovided in the masked region.

In each partial region of the mask array, an area occupied by anaperture may be greater than or equal to an area corresponding to a setaperture ratio.

A masking pattern of a group region may be repeated in the mask array.

The camera device may further include: a processor configured togenerate frequency information by transforming sensing informationsensed by the plurality of sensing elements to a frequency domain, togenerate deblurred frequency information by dividing the frequencyinformation by a frequency conversion result of a blur kernel, the blurkernel corresponding to a masking pattern of the mask array, and torestore a high-resolution image by inversely transforming the deblurredfrequency information to a time domain.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will be more apparent by describingcertain example embodiments, taken in conjunction with the accompanyingdrawings, in which:

FIGS. 1A and 1B illustrate an example of an imaging device according toan example embodiment;

FIG. 2 is a diagram illustrating an example in which a sensing elementreceives a ray through a lens element according to an exampleembodiment;

FIG. 3 is a diagram illustrating a relationship between a number ofsensing elements and a number of lens elements according to an exampleembodiment;

FIG. 4 illustrates a reduction in a focal length based on a structure ofa multi-lens array (MLA) in an imaging device according to an exampleembodiment;

FIG. 5 illustrates a blur kernel based on a structure of an MLA in animaging device according to an example embodiment;

FIG. 6 illustrates a blur kernel of an imaging device including a maskarray according to an example embodiment;

FIG. 7 is a cross-sectional view of an imaging device in which a maskarray is disposed according to an example embodiment;

FIG. 8 illustrates a design of a masking pattern of a mask arrayaccording to an example embodiment;

FIGS. 9 and 10 illustrate examples of mask arrays according to anexample embodiment;

FIG. 11 illustrates an arrangement of masking patterns for each sensingelement group in an image sensor according to an example embodiment;

FIGS. 12A and 12B illustrate examples of an arrangement of a mask arrayaccording to an example embodiment;

FIG. 13 is a block diagram illustrating a configuration of an imagingdevice according to an example embodiment;

FIG. 14 is a block diagram illustrating a configuration of an electronicterminal according to an example embodiment; and

FIGS. 15 and 16 are diagrams illustrating examples of a device in whichan image sensor is to be implemented according to example embodiments.

DETAILED DESCRIPTION

Hereinafter, some example embodiments will be described in detail withreference to the accompanying drawings. However, various alterations andmodifications may be made to the example embodiments. Here, the exampleembodiments are not construed as limited to the disclosure and should beunderstood to include all changes, equivalents, and replacements withinthe idea and the technical scope of the disclosure.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not to be limiting of the exampleembodiments. As used herein, the singular forms “a”, “an”, and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises/comprising” and/or “includes/including” when used herein,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components and/or groups thereof.

Unless otherwise defined, all terms including technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which example embodiments belong. It willbe further understood that terms, such as those defined in commonly-useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

When describing the example embodiments with reference to theaccompanying drawings, like reference numerals refer to like constituentelements and a repeated description related thereto will be omitted. Inthe description of example embodiments, detailed description ofwell-known related structures or functions will be omitted when it isdeemed that such description will cause ambiguous interpretation of thedisclosure. The drawings may not be to scale, and the relative size,proportions, and depiction of elements in the drawings may beexaggerated for clarity, illustration, and convenience.

FIGS. 1A and 1B illustrate a structure of an imaging device according toan example embodiment. FIG. 1A is a perspective view of the imagingdevice, and FIG. 1B is a cross-sectional view of the imaging device.

An imaging device 100 may include a lens array 110 and an image sensor120. The lens array 110 may include lens elements, and the image sensor120 may include optical sensing elements. The lens elements may bearranged along a plane of the lens array 110, and the optical sensingelements may be arranged along a plane of a sensing array 121 in theimage sensor 120. The plane of the lens array 110 may be placed parallelto the plane of the sensing array 121. The lens array 110 may be amulti-lens array (MLA) for imaging, and may also be referred to as an“imaging lens array.”

The term “optical sensing element”, hereinafter referred to as a“sensing element”, used herein may be an element that senses opticalinformation based on light incident on the element, and may output avalue indicating an intensity of incident light. The optical sensingelement may include, for example, a complementarymetal-oxide-semiconductor (CMOS), a charge-coupled device (CCD), and/ora photodiode.

The term “picture element” that is a pixel may be basic unit informationconstituting an image and may indicate optical information obtained bysensing light reflected from a physical position on a subjectcorresponding to a pixel position using a sensing element. The pixelposition may be a position of a pixel in an image and may conform to apixel coordinate system, and the physical position may conform to aworld coordinate system.

For reference, pixels constituting a color image may have a plurality ofcolor values (for example, a red value, a green value, and a blue valuein an RGB color system) for a single pixel position. A unit pixel of adisplay in a display field may include sub-pixels (for example, a redsub-pixel, a green sub-pixel, and a blue sub-pixel in an RGB colorsystem) for a plurality of colors to represent color values of a singlepixel position. Unlike the display field, in an image sensor field,generally, a pixel is not divided into sub-pixels for each color andrefers to a sensing element (for example, a photodiode with a front endin which a color filter is disposed) that senses one color value. Also,in the image sensor field, the term “pixel” may refer to both a singlesensing element and a value sensed by the sensing element. However, forclarity of description in example embodiments, the term “pixel” is usedherein to indicate basic unit information constituting an image, and theterm “sensing element” refers to a hardware element that outputs a pixelvalue of a pixel in response to light received from a subject, andaccordingly meanings of the pixel and the sensing element may bedistinguished.

An example in which each pixel is represented by a single sensingelement is mainly described in the following description, however,example embodiments are not limited thereto. For example, a single pixelmay be represented by a plurality of sensing elements. A plurality ofsensing elements grouped to represent a single pixel may be referred toas a “sensing element group”. Although an amount of light that may besensed by a single sensing element is limited, a sensitivity may beenhanced by representing a single pixel using values sensed by aplurality of sensing elements. An example in which a single pixel valueis sensed by a sensing element group including four sensing elementswill be described below with reference to FIG. 11 .

The image sensor 120 may include the sensing array 121, an opticalfilter 122, and a condensing lens array 123. However, this is merely anexample. Instead of the optical filter 122, an individual condensingmicro-lens 123 a of the condensing lens array 123 may have an opticalcharacteristic of transmitting a predetermined wavelength band andblocking the remaining wavelength bands other than the predeterminedwavelength band. In this case, the optical filter 122 may be omitted.

The condensing lens array 123 may include a plurality of condensingmicro-lenses 123 a configured to concentrate light passing through thelens array 110 onto the sensing array 121. For example, the condensinglens array 123 may include the same number of condensing micro-lenses123 a as the number of sensing elements included in the sensing array121. The plurality of condensing micro-lenses 123 a may be arrangedbetween an imaging optical lens and the sensing array 121, toconcentrate and transmit light passing through the imaging optical lensto a sensing element 121 a corresponding to each condensing micro-lens123 a. For example, as illustrated in FIG. 1B, a condensing micro-lens123 a may be disposed above each sensing element 121 a of the sensingarray 121 to concentrate light onto a sensing element 121 a locatedbelow the condensing micro-lens 123 a. Also, as illustrated in FIG. 1B,a color filter 122 a may be disposed between the condensing micro-lens123 a and the sensing element 121 a.

The optical filter 122 may be a filter having an optical characteristicof transmitting a predetermined wavelength band and blocking theremaining wavelength bands. For example, the optical filter 122 may beimplemented as a color filter array (CFA) including a plurality of colorfilters arranged along a filter plane. Each color filter 122 a may be afilter that allows light of a wavelength band corresponding to apredetermined color to pass and that blocks light of the remainingwavelength bands. The color filter 122 a may include, for example, ared-pass filter, a green-pass filter, and a blue-pass filter. Thered-pass filter may allow light of a wavelength band corresponding tored to pass and may block light of the remaining wavelength bands. Thegreen-pass filter may allow light of a wavelength band corresponding togreen to pass and may block light of the remaining wavelength bands. Theblue-pass filter may allow light of a wavelength band corresponding toblue to pass and may block light of the remaining wavelength bands. Inthe color filter array, color filters that individually transmit colorlight may be arranged in a Bayer pattern or other patterns along thefilter plane. The optical filter 122 may also be an infrared cut-offfilter that blocks infrared rays while passing visible rays.

A quality of an image captured and restored by the image sensor 120 maybe determined based on the number of sensing elements included in thesensing array 121 and an amount of light incident on the sensing element121 a. For example, a resolution of the image may be determined based onthe number of sensing elements included in the sensing array 121, and asensitivity of the image may be determined based on the amount of lightincident on the sensing element 121 a. The amount of light incident onthe sensing element 121 a may be determined based on a size of thesensing element 121 a. When the size of the sensing element 121 aincreases, the amount of incident light may increase, and a dynamicrange of the sensing array 121 may increase. Accordingly, when thenumber of sensing elements included in the sensing array 121 increases,a resolution of an image captured by the image sensor 120 may increase.As the size of the sensing element 121 a increases, the image sensor 120may operate more effectively in capturing a high-sensitivity image in alow-illuminance environment.

An individual lens element 111 of the lens array 110 may cover apredetermined sensing region 129 of the sensing array 121 correspondingto a lens size of the individual lens element 111. The sensing region129 covered (or substantially covered) by the lens element 111 in thesensing array 121 may be determined based on a lens size of the lenselement 111. The sensing region 129 may indicate a region in the sensingarray 121 in which rays of a predetermined field of view (FOV) arriveafter passing through the corresponding lens element 111. A size of thesensing region 129 may be represented by a diagonal length or a distancefrom a center of the sensing region 129 to an outermost point. In otherwords, light passing through the individual lens element 111 may beincident on sensing elements included in the sensing region 129.

Each of the sensing elements of the sensing array 121 may generatesensing information based on rays passing through lenses of the lensarray 110. For example, the sensing element 121 a may sense a value ofan intensity of light received through the lens element 111 as sensinginformation. The imaging device 100 may determine intensity informationcorresponding to an original signal associated with points included in aview of the imaging device 100 based on the sensing information outputby the sensing array 121 and may restore a captured image based on thedetermined intensity information.

Also, the sensing element 121 a may generate, as sensing information, acolor intensity value of a corresponding color by sensing light passingthrough the color filter 122 a. Each of the plurality of sensingelements included in the sensing array 121 may be disposed to sense acolor different from that sensed by a neighboring sensing element thatis disposed spatially adjacent thereto.

When the diversity of sensing information is sufficiently secured and afull rank relationship is formed between the sensing information andoriginal signal information corresponding to the points included in theview of the imaging device 100, a captured image corresponding to amaximum resolution of the sensing array 121 may be obtained. Thediversity of the sensing information may be secured based on parametersof the imaging device 100 such as the number of lenses included in thelens array 110 and the number of sensing elements included in thesensing array 121.

In a structure of an MLA for imaging, the imaging optical lens and thesensing array 121 may be arranged based on a fractional alignmentstructure. For example, the fractional alignment structure may representa structure in which the sensing region 129 covered by the individuallens element 111 includes a non-integer number of sensing elements.

When the lens elements included in the lens array 110 have the same lenssize, the number of lens elements included in the lens array 110 and thenumber of sensing elements included in the sensing array 121 may be in arelatively prime relationship. A ratio P/L between a number L of lenselements corresponding to one axis of the lens array 110 and a number Pof sensing elements corresponding to one axis of the sensing array 121may be determined to be a real number. Each of the lens elements maycover the same number of sensing elements as pixel offsets correspondingto P/L. For example, the sensing region 129 of FIG. 1A, which may becovered (or substantially covered) by the individual lens element 111,may include “2.3” (=7/3) sensing elements along a vertical axis and“3.67” (=11/3) sensing elements along a horizontal axis. Also, the lenselement 111 may cover a plurality of non-integer condensing micro-lenses123 a. Accordingly, in the image sensor 120, the number of condensingmicro-lenses 123 a may be the same as the number of sensing elements ofthe sensing array 121. Also, the number of lens elements (for example,imaging optical lenses) of the lens array 110 may be less than thenumber of condensing micro-lenses 123 a.

Through the fractional alignment structure as described above, anoptical center axis (OCA) of each lens element 111 in the imaging device100 may be slightly differently arranged with respect to the sensingarray 121. In other words, the lens element 111 may be disposed to beeccentric to the sensing element 121 a. Accordingly, each lens element111 of the lens array 110 may receive different light field (LF)information. LF information received by the fractional alignmentstructure will be further described below with reference to FIG. 2 .

FIG. 2 is a diagram illustrating an example in which a sensing elementreceives a ray through a lens element according to an exampleembodiment.

An LF may refer to a field indicating a direction and intensity of raysthat are radiated from an arbitrary target point and reflected from anarbitrary point on a subject. LF information may be information obtainedby combining a plurality of LFs. Since a direction of a chief ray ofeach lens element also varies, the sensing regions may receive differentLF information. Thus, an imaging device may obtain an optically greateramount of sensing information.

As illustrated in FIG. 2 , a sensing array 220 may receive and detectrays corresponding to individual points 230, for example, X1 throughX10. A plurality of rays emitted from each of the individual points 230may form LFs. Rays emitted from a first point X1 may form a first LF andmay be incident on a first sensing element S1, a fourth sensing elementS4, and a seventh sensing element S7. Rays emitted from remaining pointsX2 through X10 may also form corresponding LFs. The individual points230 may be points on a predetermined object, for example, a subject.Rays emitted from the individual points 230 may be rays such as sunlightreflected from an object. As a cross-sectional view illustrating anexample of an imaging device, FIG. 2 illustrates a lens array 210including three lens elements along one axis and the sensing array 220including ten sensing elements S1 through S10 for convenience ofdescription. However, example embodiments are not limited thereto.

The sensing elements S1 through S10 may sense rays that pass through aplurality of lens elements and that overlap with each other. The sensingelement S1 may generate overlapping sensing information, for example, anintensity value, of the rays emitted from the points X1 through X3.Similarly, the sensing elements S2 through S10 may also generateoverlapping sensing information of the rays emitted from the pluralityof individual points 230. An image sensor may restore the overlappingsensing information.

The sensing information generated by the sensing elements S1 through S10shown in FIG. 2 may be modeled as original signal information, forexample, an intensity value, corresponding to a ray incident from eachof the points 230 according to Equation 1 shown below.S=T·X  [Equation 1]

In Equation 1, S denotes a matrix indicating sensing information, forexample, a detected intensity value, sensed by individual sensingelements. X denotes a matrix indicating a signal value, for example, acolor intensity value of an incident ray, corresponding to rays incidenton the sensing elements S1 through S10 from individual points. T denotesa transformation matrix, and may indicate a relationship between thesensing information sensed by the sensing elements S1 through S10 andsignal information corresponding to incident light. In the structureshown in FIG. 2 , the rays corresponding to the individual points X1through X10, the lens elements, and the sensing elements S1 through S10may be modeled as shown in Equation 2 below. In Equation 2, theindividual points X1 through X10 may be modeled as being located atinfinite focal points from the image sensor. Distances between the imagesensor and each of the individual points X1 through X10 may be greaterthan a threshold distance.

$\begin{matrix}{\begin{bmatrix}{S\; 1} \\{S\; 2} \\{S\; 3} \\{S\; 4} \\{S\; 5} \\{S\; 6} \\{S\; 7} \\{S\; 8} \\{S\; 9} \\{S\; 10}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 \\1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 \\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 \\0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1\end{bmatrix} \cdot \begin{bmatrix}{X\; 1} \\{X\; 2} \\{X\; 3} \\{X\; 4} \\{X\; 5} \\{X\; 6} \\{X\; 7} \\{X\; 8} \\{X\; 9} \\{X\; 10}\end{bmatrix}}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

In Equation 2, for convenience of description, ray signal information,for example, ray intensity values, corresponding to the individualpoints X1 through X10 are denoted by X1 through X10. In addition,sensing information, for example, sensing intensity values, sensed bythe sensing elements S1 through S10 are denoted by S1 through S10. Arelationship, for example, the aforementioned transformation matrix,between the sensing information corresponding to the sensing elements S1through S10 included in the sensing array 220 and original signalscorresponding to the rays incident from the individual points X1 throughX10 may be determined based on an arrangement of the lens elements andthe sensing elements, the number of lens elements included in the lensarray 210, and/or the number of sensing elements S1 through S10 includedin the sensing array 220.

Equation 2 corresponds to a case in which the individual points X1through X10 are infinite focal points from the image sensor. When theindividual points X1 through X10 are located at finite focal points fromthe image sensor, an original signal received in each sensing elementmay vary depending on a distance between a subject and the image sensorand/or a geometric structure of the image sensor.

As described above, the imaging device may acquire a plurality oflow-resolution input images based on a variety of acquired sensinginformation, and may restore an output image with a higher resolutionthan that of the low-resolution input images from the low-resolutioninput images. A method of generating a single image by rearranging aplurality of low-resolution input images will be described below withreference to FIG. 3 .

FIG. 3 is a diagram illustrating a relationship between a number ofsensing elements and a number of lens elements according to an exampleembodiment.

As described above, an imaging optical lens and a sensing array may bearranged in a fractional alignment structure. FIG. 3 illustrates anexample in which a ratio P/L between a number L of lens elements and anumber P of sensing elements is 10/3.

Based on the aforementioned geometric structure of the lens array andthe sensing array, sensing elements covered by each lens element mayreceive LF information different from LF information sensed by a sensingelement covered by another lens element. For example, in the structureof FIG. 2 , the first sensing element S1 may receive LF informationincluding a combination of a first LF of the first point X1, a second LFof the second point X2, and a third LF of the third point X3. On theother hand, in the structure of FIG. 2 , a second sensing element S2neighboring the first sensing element S1 may receive LF informationincluding a combination of a fourth LF, a fifth LF, and a sixth LF. Asdescribed above, each sensing element may receive LF informationdifferent from LF information sensed by another sensing element.

To restore an image with a high resolution, an imaging device and/or animage sensor may rearrange image pixel positions of pixels indicatingthe same point or neighboring points on a subject in a plurality ofcaptured low-resolution images, based on a correlation between LFinformation.

For example, if an image is a color image, the color image may havecolor values based on a color system as pixel values, but it may bedifficult for the image sensor to simultaneously sense three colors at asingle point due to a physical limitation. Generally, a color filtercapable of allowing only one color to pass is disposed in a front end ofa sensing element, and accordingly a color sensible at a position ofeach of sensing elements may be different from a color sensed by aneighboring sensing element. Accordingly, with respect to a firstsensing element (for example, a sensing element with a front end inwhich a blue-pass filter is disposed) at a predetermined position, theimaging device and/or the image sensor may interpolate a color value(for example, a red value) that is not sensed by the first sensingelement by using a color value sensed by a second sensing element (forexample, a sensing element with a front end in which a red-pass filteris disposed) adjacent to the first sensing element. The imaging deviceand/or the image sensor may obtain three color channel images byperforming interpolation for each color channel. However, theinterpolation of the color value described above is merely an example,and other methods may also be performed depending on a design.

The imaging device and/or the image sensor may rearrange pixels for eachcolor channel, which will be described below. For example, in an RGBcolor system, the imaging device and/or the image sensor may restore ahigh-resolution red channel image by rearranging pixels oflow-resolution red channel images. Similarly, the imaging device and/orthe image sensor may restore a high-resolution blue channel image and ahigh-resolution green channel image. Thus, the imaging device and/or theimage sensor may obtain a high-resolution color image. However, exampleembodiments are not limited thereto. For example, the imaging deviceand/or the image sensor may obtain low-resolution color images bymerging the three color channel images obtained through theinterpolation as described above, and may restore a high-resolutioncolor image by rearranging pixels of the low-resolution color images.

The imaging device and/or the image sensor may construct pixelinformation of a high-resolution image by rearranging pixel positions ofpixels corresponding to sensing elements that receive similar LFinformation to be adjacent to each other. As described above, eachsensing element may receive LF information in which a plurality of LFsoverlap. When the number of same LFs included in pieces of informationsensed by two sensing elements increases, a correlation between thepieces of information may increase. A rearrangement of pixel positionsof the pixels may be performed based on a depth at which a correspondingpixel is captured. In an example, the depth at which the pixel iscaptured may be set to an arbitrary depth value, estimated throughstereo image matching, or measured by a depth sensor. In anotherexample, the pixel positions may also be rearranged by a neural networkdesigned to rearrange the pixel positions based on a depth at which asubject is captured even if the depth at which the pixel is captured isnot measured and/or estimated. The aforementioned rearrangement of thepixel positions may also be referred to as a “pixel shuffle”. Forexample, a neural network designed to output a single high-resolutionoutput image in response to an input of a plurality of low-resolutioninput images may be used to rearrange the pixel positions. The neuralnetwork may be trained based on a training data set obtained bycapturing a subject at various depths.

The image sensor may assume that points on a subject from which rays arereflected are located at infinite focal points and farther than athreshold distance from the image sensor, and may determine LFinformation to be sensed in each sensing element. The image sensor mayrearrange pixel positions of pixels having an output value output by asensing element that receive LFs emitted from points spatially adjacentto each other on the subject such that the pixel positions may beadjacent to each other.

For reference, the individual points X1 through X10 are illustrated inFIG. 2 in an order of being spatially adjacent to each other at aninfinite focal distance. The first point X1 may be adjacent to thesecond point X2. The second point X2 may be adjacent to the first pointX1 and the third point X3.

Among sensing elements 311 not rearranged yet in FIG. 3 , both LFinformation sensed in the first sensing element S1 and LF informationsensed in an eighth sensing element S8 may include LFs corresponding tothe second point X2 and the third point X3. Accordingly, the firstsensing element S1 and the eighth sensing element S8 may receive similarLF information. Equation 3 represents a result obtained by rearrangingpixels corresponding to similar LF information according to Equation 2above to be adjacent to each other.

$\begin{matrix}{\begin{bmatrix}{S\; 1} \\{S\; 8} \\{S\; 5} \\{S\; 2} \\{S\; 9} \\{S\; 6} \\{S\; 3} \\{S\; 10} \\{S\; 7} \\{S\; 4}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 \\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 \\1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}{X\; 1} \\{X\; 2} \\{X\; 3} \\{X\; 4} \\{X\; 5} \\{X\; 6} \\{X\; 7} \\{X\; 8} \\{X\; 9} \\{X\; 10}\end{bmatrix}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

Sensing elements 312 rearranged according to Equation 3 may be as shownin FIG. 3 . The first sensing element S1 may be covered by a first lens,and the eighth sensing element S8 may be covered by a third lens. Also,a fifth sensing element S5 may be covered by a second lens. Sincesensing information sensed in each sensing element corresponds to apixel constituting an image, the image sensor and/or the imaging devicemay rearrange pixels such that sensing information corresponding to rayspassing through different lenses may be adjacent. A reconstructed image325 may be an image in which pixel positions of pixels having sensingvalues obtained by sensing elements receiving similar LF information inlow-resolution images 321, 322, 323, and 324 captured by individuallenses are arranged to be adjacent.

FIG. 4 illustrates a reduction in a focal length based on a structure ofan MLA in an imaging device according to an example embodiment.

A volume of the imaging device may be determined by a focal length of alens element. This is because an image sensor needs to be spaced apartfrom the lens element by a distance corresponding to the focal length ofthe lens element to collect light refracted by the lens element. Thefocal length of the lens element may be determined by a FOV of theimaging device and a size of the lens element. If the FOV is fixed, thefocal length may increase in proportion to the size of the lens element.To capture an image in a predetermined FOV range, the size of the lenselement may need to increase as a size of a sensing array increases.

As described above, to increase a sensitivity of an image whilemaintaining a FOV and a resolution of the image, a volume of the imagesensor may be increased. To increase the sensitivity of the image whilemaintaining the resolution of the image, a size of each sensing elementmay need to be increased while maintaining a number of sensing elementsincluded in the sensing array, and thus the size of the sensing arraymay increase. To maintain the FOV, the size of the lens element and thefocal length of the lens element may increase as the size of the sensingarray increases, and thus the volume of the image sensor may increase.

When a size of each lens element included in a lens array decreases,that is, when a number of lenses included in the same area on the lensarray increases, a focal length of each lens element may decrease. Thus,a thin camera with a reduced thickness of the imaging device may beimplemented. As shown in FIG. 4 , a focal length f′ of each of an MLA420 may be reduced in comparison to a focal length f of a single lens410. For example, if the MLA 420 including “2×2” lenses is used insteadof the single lens 410, the focal length f′ may be equal to f/2.

However, an incident area S′ of an individual lens of the MLA 420 onwhich light is incident may be less than an incident area S of thesingle lens 410 on which light is incident. For example, the incidentarea S′ may be equal to S/4. Also, an incident solid angle Ωcorresponding to an angle range of rays incident on an individualsensing element, for example, a sensing element S2, may increase due toa decrease in the focal length. For example, an incident solid angle Ω′corresponding to an individual lens of the MLA 420 may be equal to 4Ω. Ablur kernel based on an increase in an incident solid angle will bedescribed below with reference to FIG. 5 .

FIG. 5 illustrates a blur kernel based on a structure of an MLA in animaging device according to an example embodiment.

When lenses are designed at the same FOV in a structure of an MLA 510, afocal length may decrease and an incident solid angle may increase if anumber of lenses for the same area increases as described above withreference to FIG. 4 . When the incident solid angle increases, a numberof signals overlapping in a sensing element may increase. As signalsoverlap, information sensed by the sensing element may be blurred. Forexample, when a FOV is 100 degrees and an image sensor 520 includes“100” sensing elements in an imaging device with a single lens, a FOV ofeach of the sensing elements may be 1 degree. In an example, if a numberof lenses on the same area is “2”, “50” sensing elements may be coveredby each lens of the image sensor 520. Thus, a FOV of each of the sensingelements may be 2 degrees. In another example, if a number of lenses onthe same area is “50”, “2” sensing elements may be covered by each lensof the image sensor 520, and a FOV of each of the sensing elements maybe 50 degrees. A larger amount of LF information may overlap as a FOVsensible by an individual sensing element increases, and thus a blurlevel of sensing information may increase.

A blur kernel may be a kernel obtained by modeling a blur caused byoverlapping of LF information in an individual sensing element, and mayalso be referred to as a “blur model” or a “blur filter”. LF informationof {right arrow over (x_(a))} through {right arrow over (x_(b))} may becondensed by a lens aperture S through the MLA 510, and the condensed LFinformation may be sensed by an i-th sensing element. {right arrow over(x_(a))} may represent a bundle of rays concentrated on an outermostpoint of a sensing element, and {right arrow over (x_(b))} may representa bundle of rays concentrated on an opposite outermost point of thesensing element. Here, i may be an integer between “1” and “n”,inclusive, and n may be a total number of sensing elements included inthe image sensor 520. A signal intensity of a bundle of concentratedrays may be represented as shown in Equations 4 and 5 below.

$\begin{matrix}{x_{a} = {\int\limits_{S}{\overset{\longrightarrow}{x_{a}}{dA}}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack \\{x_{b} = {\int\limits_{S}{\overset{\longrightarrow}{x_{b}}{dA}}}} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

In Equation 4, x_(a) denotes an intensity of rays concentrated in adirection a. In Equation 5, x_(b) denotes an intensity of raysconcentrated in a direction b.

The i-th sensing element may sense an intensity value obtained byaccumulating all LF information within a FOV Ω. An intensity valueobtained by accumulating all LF information within a FOV of the i-thsensing element may be discretely approximated, which may be representedas shown in Equation 6 below.

$\begin{matrix}{{s\lbrack i\rbrack} = {{\int_{\Omega = a}^{\Omega = b}{x_{\Omega}d\theta}} \approx {\sum\limits_{\Omega = a}^{\Omega = b}x_{\Omega}}}} & \left\lbrack {{Equation}6} \right\rbrack\end{matrix}$

In Equation 6, s[i] denotes an intensity value sensed by the i-thsensing element. In an imaging device configured with an imaging opticalsystem including the MLA 510, a blur kernel of an image may be modeledso that LF information within a FOV of each sensing element may overlapwith the same size, for example, “1”. Sensing information s[i] sensed bythe i-th sensing element may be modeled as a convolution relationshipbetween original LF information x_(Ω)[i] and a uniform blur kernel h[i],as shown in Equation 7 below.s[i]=(x _(Ω)[i]*h[i])+t[i]  [Equation 7]

In Equation 7, h[i] denotes a blur kernel, and t[i] denotes a noisecomponent. If a Fourier transform of Equation 7 is performed, aconvolution operation may be replaced by a multiplication, as shown inEquation 8 below.S(f)=X _(Ω)(f)·H(f)+T(f)  [Equation 8]

In Equation 8, X_(Ω)(f) denotes frequency information of the originalsignal, H(f) denotes a frequency response characteristic of a blurkernel, and T(f) denotes frequency information of noise. The frequencyinformation X_(Ω)(f) of the original signal to be restored may becalculated from Equation 8, as shown in Equation 9 below.

$\begin{matrix}{{X_{\Omega}(f)} = {\frac{1}{H(f)}\left( {{S(f)} - {T(f)}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 9} \right\rbrack\end{matrix}$

If an inverse Fourier transform is applied to Equation 9, deblurredx_(Ω)[i] may be obtained.

However, since T(f) that is a noise component in Equation 9 is anunknown component, an error due to T(f)/H(f) may occur even if TO ismodeled as a statistical probability distribution. As shown in FIG. 5 ,a noise component may be amplified due to the uniform blur kernel h[i]in a spatial domain. The frequency response characteristic H(f) foruniform blur kernel h[i] that is frequency-converted may include zerocrossing points 590, and impulse components may be caused by the zerocrossing points 590 in a reciprocal of the frequency responsecharacteristic H(f). Since the impulse components are multiplied by thenoise component, noise may be greatly amplified in a deblurring process.

FIG. 6 illustrates a blur kernel of an imaging device including a maskarray according to an example embodiment.

A blur kernel h′[i] of the imaging device may be designed to suppress anoise component. For example, the blur kernel h′[i] may be designed sothat zero crossing points 690 may be minimized in a frequency responsecharacteristic H′(f) of a frequency domain.

The mask array may be disposed between an imaging lens array and asensing element, and may include mask elements that block light directedin a portion of directions. For example, the mask array may block lightdirected in a portion of directions among light passing through an MLAand may selectively allow light directed in other directions to pass, toform the blur kernel h′[i].

FIG. 7 is a cross-sectional view of an imaging device in which a maskarray is disposed according to an example embodiment.

The imaging device may include an imaging lens array 710 and an imagesensor 720. The imaging lens array 710 may include imaging opticallenses configured to transmit light received from an outside of theimaging device, and may be disposed as shown in FIG. 1 above. Theimaging optical lenses may form an imaging optical system.

The imaging optical system may be an optical system that performsimaging on a sensing array 721, and an optical characteristic may bedetermined by, for example, a focal length, a size, a shape and astructure of the imaging lens array 710, and/or a geometricalrelationship between the imaging lens array 710 and the sensing array721. For example, the imaging optical system may further include ablocking portion 711 configured to prevent light passing through anindividual imaging optical lens from reaching a sensing region coveredby another imaging optical lens. Also, the imaging optical system mayfurther include an aperture (not shown) configured to transmit light toan imaging optical lens.

The sensing array 721 may include a plurality of sensing elementsconfigured to sense light received from the outside. The sensingelements may each receive light directed in a plurality of directions,and a light bundle incident on the imaging device in a single directionmay be concentrated as rays in the direction by an imaging optical lens.A sensing element belonging to an arbitrary sensing region may receiverays that respectively correspond to the plurality of directions andthat are concentrated by an imaging optical lens that covers the sensingregion. In FIG. 7 , a first ray 791 of a first direction and a secondray 792 of a second direction which are directed toward sensing elementsby passing through an imaging optical lens are illustrated as examples.

Since the imaging optical system includes the imaging lens array 710with a multi-lens structure, as described above, an individual sensingelement of the sensing array 721 may sense rays that are received inmultiple directions and that overlap with each other. In other words,overlapping of light imaged in the sensing array 721 may be modeled asthe blur kernel h′ [i] of FIG. 6 .

A mask array 724 may include a plurality of mask elements and may bedisposed on the sensing array 721. The mask elements may be disposedabove a position in which a sensing element of the sensing array 721 isdisposed, and may absorb and block a portion or all of light directed tothe position. The mask array 724 may modify an imaging optical systemcorresponding to the blur kernel of FIG. 5 to an optical systemcorresponding to the blur kernel h′ [i] of FIG. 6 . The mask array 724may also be understood as transmitting light passing through an imagingoptical lens to the sensing array 721 by coding the light, and may bereferred to as the “coded mask array 724”. The mask array 724 may bedisposed at a position inside the sensing array 721 or a position incontact with the sensing array 721. Although it is desirable that,regardless of a position of the mask array 724, there is no gap betweenthe mask array 724 and sensing elements (for example, a first sensingelement 721 a through a fifth sensing element 721 e), the mask array 724may be spaced apart from the sensing elements by about 1 micrometer (μm)or less due to a limitation of a manufacturing process.

The plurality of mask elements may be disposed in a pattern in whichzero crossing of a frequency response characteristic of a blur kernel isminimized. For example, the first ray 791 passing through the imagingoptical lens may be incident on a sensing element through mask elements.The second ray 792 may be incident on a mask element and may beabsorbed. The second ray 792 may be a ray that causes zero crossing inthe frequency response characteristic of the blur kernel described abovewith reference to FIG. 6 . Thus, the mask array 724 may block rays in adirection causing zero crossing.

The imaging device and/or the image sensor 720 may sense overlapping LFinformation in a direction selectively filtered through the mask array724. Thus, the imaging device and/or the image sensor 720 including themask array 724 may restore an image with reduced noise. While a noisemay be caused by a loss of an amount of light due to the mask array 724,a noise suppression effect by a blur kernel modified due to the maskarray 724 may be greatly increased. As a result, a quality of therestored image may be enhanced by a masking pattern of the mask array724.

The plurality of mask elements may be formed and/or disposed accordingto the masking pattern. The masking pattern may be repeated in units ofsensing elements or in units of sensing element groups. A first maskingpattern disposed on the first sensing element 721 a, a second maskingpattern disposed on a second sensing element 721 b, a third maskingpattern disposed on a third sensing element 721 c, a fourth maskingpattern disposed on a fourth sensing element 721 d, and a fifth maskingpattern disposed on the fifth sensing element 721 e may be the same.

A design of the masking pattern of the mask array 724 will be describedbelow with reference to FIG. 8 .

FIG. 8 illustrates a method of designing a masking pattern of a maskarray according to an example embodiment.

A plurality of mask elements may be formed in a pattern in which a costfunction determined based on a frequency response characteristic of afilter is minimized. A masking pattern disposed for each sensing elementor for each sensing element group in the mask array may be determined asa pattern in which a Euclidean norm of 1/H(f) that is a reciprocal of afrequency response characteristic of a blur kernel is minimized.

Referring to FIG. 8 , in operation 810, a target aperture ratio of themask array may be set. An aperture ratio may be a ratio of light to betransmitted with respect to light incident on a pattern regioncorresponding to the masking pattern, and may represent a ratio of anarea of an open region with respect to an area of the pattern region.The pattern region may be a region corresponding to a sensing element ora sensing element group in the mask array. The aperture ratio may be setto, for example, a ratio of 10% to 90%, or a ratio of 30% to 70%.Desirably, the aperture ratio may be set to a ratio of 40% to 60%.

In operation 820, a masking pattern according to the set aperture ratiomay be generated. The mask array may be segmented into pattern regionscorresponding to individual sensing elements, and each of the patternregions may be segmented into a plurality of spaces. An individual spacemay define a unit region in which a mask element may be formed. A spacein which a mask element is formed may be referred to as a “closed space”and a space in which a mask element is not formed may be referred to asan “open space”. In other words, a closed space in the mask array mayabsorb all or a portion of light, and an open space may allow light topass. Masking patterns in which open spaces and closed spaces arecombined based on the target aperture ratio set in operation 810 may begenerated. If a pattern region includes “N×N” spaces and the targetaperture ratio is 50%, “N²/2” masking patterns may be generated. Closedspaces may be used to classify each of mask elements according to atransmittance thereof (or a transmission level), which will be describedbelow with reference to FIG. 10 .

An example of generating an entire combination of the mask patterns isillustrated in operation 820 for convenience of description, however,example embodiments are not limited thereto. In an example, pre-definedpatterns may be combined, or a masking pattern to be searched for may begenerated by swapping or toggling between patterns within givencombinations. For example, if a condition that patterns are symmetric isadded, a pattern may be searched for with respect to repeated regionsonly, and a generated pattern may be used symmetrically.

In operation 830, a cost function value may be calculated for eachgenerated masking patterns. A cost function based on a reciprocal of thefrequency response characteristic of the blur kernel described abovewith reference to FIGS. 5 and 6 may be used. Due to the mask elementsformed based on the masking pattern, a blur kernel of an imaging opticalsystem may be modified, and a frequency response characteristic H′(f) ofthe modified blur kernel may be calculated. A cost function based on areciprocal 1/H′(f) of the frequency response characteristic H′(f) of themodified blur kernel may be determined based on a Euclidean norm of areciprocal of a function representing a frequency responsecharacteristic, a variance of the reciprocal, and/or a reciprocal of aEuclidean norm of the function. For example, a cost function E shown inEquation 10 below may be used

$\begin{matrix}{E = {{\alpha \cdot {\frac{1}{H^{\prime}(f)}}_{2}} + {\beta \cdot \frac{1}{{{H^{\prime}(f)}}_{0}}} + {\gamma \cdot {{var}\left( \frac{1}{H^{\prime}(f)} \right)}}}} & \left\lbrack {{Equation}\mspace{14mu} 10} \right\rbrack\end{matrix}$

In Equation 10,

${\frac{1}{H^{\prime}(f)}}_{2}$denotes the Euclidean norm of the reciprocal of the frequency responsecharacteristic H′(f). A number of zero crossing points may be minimizedby

${\frac{1}{H^{\prime}(f)}}_{2}\frac{1}{.{{H^{\prime}(f)}}_{0}}$denotes a reciprocal of a Euclidean norm of the frequency responsecharacteristic H′(f). By

$\frac{1}{{{H^{\prime}(f)}}_{0}},$a value of the frequency response characteristic H′(f) may be designednot to be zero.

${var}\left( \frac{1}{H^{\prime}(f)} \right)$denotes a variance of the reciprocal of the frequency responsecharacteristic H′(f). 1/H′ (f) may be designed so that the variance maybe minimized in a frequency domain. The cost function E of Equation 10may be a weighted average value of individual cost factors, and α, β,and γ denote weights.

In operations 840 and 850, a masking pattern in which a cost functionvalue is minimized may be searched for and determined. Cost functionvalues for each of masking patterns with combinations of closed spacesand open spaces may be calculated in a given condition, and a maskingpattern in which noise amplification is minimized by a cost function maybe searched for. The masking pattern found as a result of searching maybe determined as a pattern of the mask array in operation 850. An imagesensor may include a mask array in which the found masking pattern isrepeated.

FIGS. 9 and 10 illustrate examples of mask arrays according to anexample embodiment.

FIG. 9 illustrates an example in which a transmission state of light foreach individual space of a mask array 924 is classified as a binarystate. In other words, an open space 991 may allow all light incidentthereon to pass, and a closed space 992 may absorb all light incidentthereon.

The mask array 924 may be segmented into a plurality of group regions. Agroup region 924 a may be a region in the mask array 924 which covers asingle sensing element or a plurality of sensing elements of aneighboring sensing array 921. Although an example in which the groupregion 924 a of the mask array 924 covers a single sensing element isillustrated in FIG. 9 , the group region 924 a of the mask array 924 maycover a sensing element group including a plurality of sensing elementsgrouped to represent a single pixel, which will be described below withreference to FIG. 11 . A masking pattern 990 of the group region 924 amay be repeated. For example, all the plurality of group regions of themask array 924 may have the same masking pattern 990.

A pattern region of the mask array 924 corresponding to an individualsensing element may include an aperture region and a masked region. Theaperture region may occupy an area corresponding to an aperture ratiowith respect to a total area of a corresponding region, and the maskedregion may occupy the remaining area. For example, the masking pattern990 may be a pattern designed at a target aperture ratio of 50%. Themasking patterns 990 may be divided into a total of “7×7=49” spaces andmay include “24” closed spaces 992 and “25” open spaces.

Also, the masking pattern 990 may secure an aperture ratio of a partialregion 995 as well as an aperture ratio relative to the total area. Anarea occupied by an aperture part in the partial region 995 may begreater than or equal to an area corresponding to a set aperture ratio.The partial region 995 of the masking pattern 990 may include “4×4=16”spaces in total including “8” closed spaces 992 and “8” open spaces 991,and accordingly the aperture ratio may be 50%. Even when a partialregion including “4×4=16” spaces at another position of the maskingpattern 990 is considered, the aperture ratio may be 50%. In otherwords, the masking pattern 990 may be designed so that regions with atarget aperture ratio may be uniformly distributed.

In addition, a number of spaces included in a region corresponding to anindividual masking pattern 990 may be greater than or equal to a numberof imaging optical lenses of an imaging optical system. As describedabove with reference to FIG. 5 , information sensed by a sensing elementmay be blurred in proportion to a number of lenses in an MLA. To restorean image with a high resolution by offsetting a blur level proportionalto the number of lenses, a pattern region may include spacescorresponding to the number of lenses to provide a deblurring function.When a lens array 910 includes “7×7=49” imaging optical lenses, themasking pattern 990 may include at least “7×7=49” spaces. Informationtransmitted to the sensing array 921 through the lens array 910 may beblurred by 1/49, and the masking pattern 990 may provide a correspondingdeblurring capacity of 49 times.

FIG. 10 illustrates an example in which an individual mask element of amask array has one of two or more transmission levels. A mask elementmay block and/or absorb a portion of light that reaches the maskelement, and a transmission level may indicate a level and/or aproportion of transmitting incident light. A mask element may be formedand/or disposed along a plane of the mask array based on theabove-described masking pattern.

An individual mask element may be segmented into a plurality of regions,and a transmittance of the individual mask element may be determinedbased on a ratio of an open region and a closed region among theplurality of regions. FIG. 10 illustrates a masking pattern 1000designed with five transmission levels. Each mask element may besegmented into four regions with the same size. A space 1010 in which amask element is not formed may have a first transmission level (forexample, 100% transmission). A first mask element 1021 may have a secondtransmission level (for example, 75% transmission), and may include asingle closed space. A second mask element 1022 may have a thirdtransmission level (for example, 50% transmission), and may includeclosed regions corresponding to half (for example, two regions) ofregions. A third mask element 1023 may have a fourth transmission level(for example, 25% transmission), and may include three closed regions. Afourth mask element 1024 may have a fifth transmission level (forexample, 0% transmission) and all regions of the fourth mask element1023 may be closed regions. An aperture ratio determined based on anumber of open regions and a number of closed regions or an area of anopen region and an area of a closed region may be interpreted as a ratioof an amount of light to be transmitted to an amount of light incidenton a region.

FIG. 11 illustrates an arrangement of masking patterns for each sensingelement group in an image sensor according to an example embodiment.

An example in which a single sensing element is covered by each groupregion of a mask array is illustrated in FIG. 9 , and an example inwhich a sensing element group 1121 including a plurality of sensingelements 1121 a is covered by each group region 1141 of a mask array1140 is illustrated in FIG. 11 . In the example of FIG. 11 , a pixelvalue of an image pixel may be determined based on values sensed by thesensing element group 1121 (for example, “2×2” sensing elements 1121 a).The mask array 1140 may include a plurality of masking patterns. Asingle masking pattern may be disposed per sensing element as shown inFIG. 9 , while a single masking pattern 1190 may be disposed per sensingelement group 1121 as shown in FIG. 11 . All the masking patterns of themask array 1140 may have the same shape.

FIGS. 12A and 12B illustrate examples of an arrangement of a mask arrayaccording to an example embodiment.

A mask array may be disposed at various positions that allow lightdirected in a predetermined direction in a sensor for a camera to beblocked and/or overlap. Referring to FIG. 12A, a mask array 1240 a maybe disposed on a color filter 1220 between a condensing lens array 1230and a sensing array 1210. Referring to FIG. 12B, a mask array 1240 b maybe disposed between a color filter 1220 and a sensing array 1210. Asdescribed above, the mask array 1240 b and the sensing array 1210 may bespaced apart from each other by 1 μm or less.

FIG. 13 is a block diagram illustrating a configuration of an imagingdevice according to an example embodiment.

An imaging device 1300 may include a lens array 1310 and an imagesensor.

The lens array 1310 may include imaging optical lenses configured totransmit light received from the outside.

The image sensor may be a sensor that senses light passing through thelens array 1310. The image sensor may include a mask array 1324, asensing array 1321, and a processor 1330. The mask array 1324 and thesensing array 1321 have been described above with reference to FIGS. 1through 12B, and accordingly further description thereof is not repeatedherein.

The processor 1330 may restore an image based on sensing informationsensed by sensing elements. The processor 1330 of the image sensor mayalso be referred to as, for example, an image signal processor (ISP).The processor 1330 may generate frequency information by transformingsensing information to a frequency domain, and may generate deblurredfrequency information by dividing the frequency information by afrequency conversion result of a blur kernel corresponding to a maskingpattern of the mask array 1324. The processor 1330 may restore ahigh-resolution image by inversely transforming the deblurred frequencyinformation to a time domain. The sensing information may be used in,for example, depth estimation for a subject, refocusing, dynamic rangeimaging, and capturing a high-sensitivity image in a low-illuminanceenvironment, in addition to image restoration.

FIG. 14 is a block diagram illustrating a configuration of an electronicterminal according to an example embodiment.

An electronic terminal 1400 may include an imaging module 1410 and aprocessor 1420.

The imaging module 1410 may include a lens array 1411 and an imagesensor. The image sensor may include a mask array 1412 and a sensingarray 1413. Unlike the processor 1330 included in the image sensor asshown in FIG. 13 , FIG. 14 illustrates that a processor is locatedindependently of the image sensor. Since the lens array 1411, the imagesensor, and the processor 1420 have been described above, furtherdescription is not repeated herein. The processor 1420 of FIG. 14 may bean application processor (AP).

FIGS. 15 and 16 are diagrams illustrating examples of a device in whichan image sensor is to be implemented according to example embodiments.

An image sensor and/or imaging device may be applied to varioustechnical fields. A lens array including a plurality of lenses, and asensor including a plurality of sensing elements may be designed to bespaced apart from each other by a relatively short focal length, theimaging device may be implemented as an ultra-thin camera with a smallthickness and a large sensor for high-definition capturing.

The image sensor and/or imaging device may be mounted on a mobileterminal. The mobile terminal may be a movable terminal that is notfixed at any location, and may include, for example, a vehicle, anartificial intelligence speaker, and a portable device such as asmartphone, a tablet personal computer (PC) or a foldable smartphone.

As illustrated in FIG. 15 , an imaging module 1510 may be applied to afront camera or a rear camera of a smartphone. The imaging module 1510may have a structure in which a large full frame sensor and an MLA arecombined, and may be applied to a camera of a smartphone.

Also, the imaging module 1510 may be implemented in a vehicle in a thinstructure or curved structure. As illustrated in FIG. 16 , an imagingdevice 1610 may be implemented as a front camera or a rear camera havinga curved shape in a vehicle 1600. In addition, the imaging device 1610may also be applied to a field, for example, such as a digitalsingle-lens reflect (DSLR) camera, a drone, a closed-circuit television(CCTV), a webcam camera, a panoramic camera, a movie or broadcast videocamera, a virtual reality (VR)/augmented reality (AR) camera, aflexible/stretchable camera, a compound-eye camera, or a contact lenstype camera. Also, the imaging device 1610 may also be applied tomulti-frame super-resolution image restoration for increasing aresolution based on information about a plurality of captured frames.

The units described herein may be implemented using hardware componentsand software components. For example, the hardware components mayinclude microphones, amplifiers, band-pass filters, audio to digitalconvertors, non-transitory computer memory and processing devices. Aprocessing device may be implemented using one or more general-purposeor special purpose computers, such as, for example, a processor, acontroller and an arithmetic logic unit (ALU), a digital signalprocessor, a microcomputer, a field programmable gate array (FPGA), aprogrammable logic unit (PLU), a microprocessor or any other devicecapable of responding to and executing instructions in a defined manner.The processing device may run an operating system (OS) and one or moresoftware applications that run on the OS. The processing device also mayaccess, store, manipulate, process, and create data in response toexecution of the software. For purpose of simplicity, the description ofa processing device is used as singular; however, one skilled in the artwould appreciate that a processing device may include multipleprocessing elements and multiple types of processing elements. Forexample, a processing device may include multiple processors or aprocessor and a controller. In addition, different processingconfigurations are possible, such a as parallel processors.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independently orcollectively instruct or configure the processing device to operate asdesired. Software and data may be embodied permanently or temporarily inany type of machine, component, physical or virtual equipment, computerstorage medium or device, or in a propagated signal wave capable ofproviding instructions or data to or being interpreted by the processingdevice. The software also may be distributed over network coupledcomputer systems so that the software is stored and executed in adistributed fashion. The software and data may be stored by one or morenon-transitory computer readable recording mediums.

The method according to the above-described example embodiments may berecorded in non-transitory computer-readable media including programinstructions to implement various operations which may be performed by acomputer. The media may also include, alone or in combination with theprogram instructions, data files, data structures, and the like. Theprogram instructions recorded on the media may be those speciallydesigned and constructed for the purposes of the example embodiments, orthey may be of the well-known kind and available to those having skillin the computer software arts. Examples of non-transitorycomputer-readable media include magnetic media such as hard disks,floppy disks, and magnetic tape; optical media such as compact discread-only memories (CD ROMs) and digital versatile disc (DVDs);magneto-optical media such as optical discs; and hardware devices thatare specially configured to store and perform program instructions, suchas a read-only memory (ROM), a random access memory (RAM), a flashmemory, and the like. Examples of program instructions include bothmachine code, such as code produced by a compiler, and files containinghigher level code that may be executed by the computer using aninterpreter. The described hardware devices may be configured to act asone or more software modules in order to perform the operations of theabove-described example embodiments, or vice versa.

At least one of the components, elements, modules or units describedherein may be embodied as various numbers of hardware, software and/orfirmware structures that execute respective functions described above,according to an example embodiment. For example, at least one of thesecomponents, elements or units may use a direct circuit structure, suchas a memory, a processor, a logic circuit, a look-up table, etc. thatmay execute the respective functions through controls of one or moremicroprocessors or other control apparatuses. Also, at least one ofthese components, elements or units may be specifically embodied by amodule, a program, or a part of code, which contains one or moreexecutable instructions for performing specified logic functions, andexecuted by one or more microprocessors or other control apparatuses.Also, at least one of these components, elements or units may furtherinclude or implemented by a processor such as a central processing unit(CPU) that performs the respective functions, a microprocessor, or thelike. Two or more of these components, elements or units may be combinedinto one single component, element or unit which performs all operationsor functions of the combined two or more components, elements of units.Also, at least part of functions of at least one of these components,elements or units may be performed by another of these components,element or units. Further, although a bus is not illustrated in theblock diagrams, communication between the components, elements or unitsmay be performed through the bus. Functional aspects of the aboveexample embodiments may be implemented in algorithms that execute on oneor more processors. Furthermore, the components, elements or unitsrepresented by a block or processing operations may employ any number ofrelated art techniques for electronics configuration, signal processingand/or control, data processing and the like.

While this disclosure includes example embodiments, it will be apparentto one of ordinary skill in the art that various changes in form anddetails may be made in these example embodiments without departing fromthe spirit and scope of the claims and their equivalents. The exampleembodiments described herein are to be considered in a descriptive senseonly, and not for purposes of limitation. Descriptions of features oraspects in each example are to be considered as being applicable tosimilar features or aspects in other examples. Suitable results may beachieved if the described techniques are performed in a different order,and/or if components in a described system, architecture, device, orcircuit are combined in a different manner and/or replaced orsupplemented by other components or their equivalents.

Therefore, the scope of the disclosure is defined not by the detaileddescription, but by the claims and their equivalents, and all variationswithin the scope of the claims and their equivalents are to be construedas being included in the disclosure.

What is claimed is:
 1. An image sensor comprising: a mask arraycomprising a plurality of mask elements, the plurality of mask elementsbeing configured to, among light passing through imaging optical lensesand incident onto the mask array in a plurality of directions, blocklight in a first portion of the plurality of directions, and allow lightin a second portion of the plurality of directions to pass therethrough;and a sensing array comprising a plurality of sensing elements, theplurality of sensing elements being configured to sense the lightpassing through the imaging optical lenses and the mask array, whereinin each partial region of the mask array, an area occupied by anaperture is greater than or equal to an area corresponding to a setaperture ratio.
 2. The image sensor of claim 1, further comprising: acolor filter provided above the sensing array and configured to filterlight of a portion of wavelength bands from light incident on each ofthe plurality of sensing elements, wherein the mask array is providedbetween the color filter and the sensing array.
 3. The image sensor ofclaim 1, further comprising: a condensing lens array provided above thesensing array, wherein the mask array is provided between the condensinglens array and the sensing array.
 4. The image sensor of claim 1,wherein the mask array and the plurality of sensing elements are spacedapart from each other by 1 micrometer (μm) or less.
 5. The image sensorof claim 1, wherein the mask array and the plurality of sensing elementsare in contact with each other.
 6. The image sensor of claim 1, whereina first region of the mask array corresponding to a sensing element ofthe plurality of sensing elements comprises: an aperture regionoccupying an area corresponding to an aperture ratio with respect to atotal area of the first region; and a masked region occupying aremaining area of the first region, the plurality of mask elements beingprovided in the masked region.
 7. The image sensor of claim 6, whereinthe aperture ratio with respect to the total area of the first region isin a range from an aperture ratio of about 40% with respect to the totalarea of the first region to an aperture ratio of about 60% with respectto the total area of the first region.
 8. The image sensor of claim 1,wherein the mask array is segmented into a plurality of group regionscorresponding to a plurality of sensing element groups, and each of theplurality of group regions in the mask array is configured to cover asensing element group, the sensing element group comprising a pluralityof sensing elements that are grouped to represent a single pixel.
 9. Theimage sensor of claim 8, wherein a masking pattern of a group region isrepeated in the mask array.
 10. The image sensor of claim 8, wherein allof the plurality of group regions in the mask array have a same maskingpattern.
 11. The image sensor of claim 1, wherein a number of spacesincluded in a masking pattern, which is repeated in the mask array, isgreater than or equal to a number of the imaging optical lenses.
 12. Theimage sensor of claim 1, wherein the plurality of mask elements have twoor more transmission levels.
 13. The image sensor of claim 1, whereineach of the plurality of mask elements is segmented into a plurality ofregions, and a transmittance of each of the plurality of mask elementsis determined based on a ratio of an open region and a closed regionamong the plurality of regions.
 14. The image sensor of claim 1, furthercomprising: a processor configured to restore an image based on sensinginformation sensed by the plurality of sensing elements.
 15. The imagesensor of claim 1, wherein the mask array comprises a plurality ofmasking patterns, and each of the plurality of masking patterns isconfigured to cover a sensing element group, the sensing element groupcomprising two or more sensing elements in the sensing array.
 16. Acamera device comprising: an imaging lens array comprising imagingoptical lenses, the imaging optical lenses configured to transmit lightreceived from an outside of the camera device; a sensing arraycomprising a plurality of sensing elements, the plurality of sensingelements being configured to sense light passing through the imaginglens array; and a mask array comprising a plurality of mask elements,the mask array being provided between the imaging lens array and thesensing array, wherein a first region of the mask array corresponding toa sensing element of the plurality of sensing elements comprises: anaperture region occupying an area corresponding to an aperture ratiowith respect to a total area of the first region; and a masked regionoccupying a remaining area in the first region, the plurality of maskelements provided in the masked region.
 17. The camera device of claim16, wherein the mask array is provided at one of a position inside thesensing array and a position in contact with the sensing array on theplurality of sensing elements.
 18. The camera device of claim 16,wherein in each partial region of the mask array, an area occupied by anaperture is greater than or equal to an area corresponding to a setaperture ratio.
 19. The camera device of claim 16, wherein a maskingpattern of a group region is repeated in the mask array.
 20. The cameradevice of claim 16, further comprising: a processor configured togenerate frequency information by transforming sensing informationsensed by the plurality of sensing elements to a frequency domain, togenerate deblurred frequency information by dividing the frequencyinformation by a frequency conversion result of a blur kernel, the blurkernel corresponding to a masking pattern of the mask array, and torestore a high-resolution image by inversely transforming the deblurredfrequency information to a time domain.
 21. An image sensor comprising:a mask array comprising a plurality of mask elements, the plurality ofmask elements being configured to, among light passing through imagingoptical lenses and incident onto the mask array in a plurality ofdirections, block light in a first portion of the plurality ofdirections, and allow light in a second portion of the plurality ofdirections to pass therethrough; a sensing array comprising a pluralityof sensing elements, the plurality of sensing elements being configuredto sense the light passing through the imaging optical lenses and themask array; and a processor configured to restore an image based onsensing information sensed by the plurality of sensing elements, whereinthe processor is further configured to generate frequency information bytransforming the sensing information to a frequency domain, to generatedeblurred frequency information by dividing the frequency information bya frequency conversion result of a blur kernel, the blur kernelcorresponding to a masking pattern of the mask array, and to restore ahigh-resolution image by inversely transforming the deblurred frequencyinformation to a time domain.